Varifocal aberration compensation for near-eye displays

ABSTRACT

A method of operation in a near-eye display system includes determining, using an eye tracking component of the near-eye display system, a pose of a user&#39;s eye. A shift vector is determined for a magnifier lens of the near-eye display system based on the pose of the user&#39;s eye, and the shift vector is communicated to an actuator of the near-eye display system to instruct translation of the magnifier lens relative to the user&#39;s eye. After translation of the magnifier lens, an array of elemental images is rendered at a position within a near-eye lightfield frame and communicated for display at a display panel of the near-eye display system.

BACKGROUND

Head-mounted displays (HMDs) and other near-eye display systems canutilize a near-eye lightfield display or other computational display toprovide display of three-dimensional (3D) graphics. Generally, thenear-eye lightfield display employs one or more display panels and anumber of lenses, pinholes, or other optical elements that overlie theone or more display panels. A rendering system renders an array ofelemental images, with each elemental image representing an image orview of an object or scene from a corresponding perspective or virtualcamera position. Such near-eye lightfield displays typically exhibit adisparity between vergence (i.e., simultaneous movement of the eyes inopposite directions to maintain binocular fixation of eye gaze onobjects at different distances) due to the physical surface of thedisplay panel and accommodation (i.e., changing the focal power of thelenses in the eyes) due to the focal point in simulated graphics of theHMDs.

In natural viewing (as opposed to viewing a virtual scene), vergence andaccommodation requirements are consistent with one another: looking at anearer object requires convergence and an increase in lens focal power,while looking at a farther object requires divergence and a decrease infocal power. Accordingly, and because the distances to which the eyesconverge and accommodate are generally the same, the two responses arecoupled such that changes in vergence produce changes in accommodation,and vice versa. However, conventional near-eye display systems oftenencounter vergence-accommodation conflicts when distance from thedisplay panel (which is generally fixed) differs from the virtual depthof objects presented on the display panel (which generally varies withthe content), resulting in discomfort and fatigue for the viewer.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure may be better understood, and its numerousfeatures and advantages made apparent to those skilled in the art byreferencing the accompanying drawings. The use of the same referencesymbols in different drawings indicates similar or identical items.

FIG. 1 is a diagram illustrating a near-eye display system employing eyetracking and corresponding lens actuation to provide varifocalaberration compensation in accordance with some embodiments.

FIG. 2 is a diagram illustrating an example of dynamicvergence-accommodation conflict correction in the near-eye displaysystem of FIG. 1 in accordance with some embodiments.

FIG. 3 is a flow diagram illustrating a method for dynamicvergence-accommodation conflict correction in the near-eye displaysystem of FIG. 1 in accordance with some embodiments.

FIG. 4 is a diagram illustrating an example of dynamic gaze-dependentaberration correction in the near-eye display system of FIG. 1 inaccordance with some embodiments.

FIG. 5 is a flow diagram illustrating a method for dynamicgaze-dependent aberration correction in the near-eye display system ofFIG. 1 in accordance with some embodiments.

DETAILED DESCRIPTION

FIGS. 1-5 illustrate example methods and systems for dynamic varifocalaberration compensation based on user eye pose in a near-eye displaysystem. In at least one embodiment, the near-eye display system employsa computational display to display near-eye lightfield frames of imageryto a user so as to provide the user with an immersive virtual reality(VR) or augmented reality (AR) experience. Each near-eye lightfieldframe is composed of an array of elemental images, with each elementalimage representing a view of an object or scene from a differentcorresponding viewpoint.

To provide improved depth cues without a corresponding reduction in FOVand/or perceived frame-rate, in at least one embodiment the near-eyedisplay systems described herein utilize a dynamic magnifier lensshifting technique wherein an eye tracking component is utilized todetermine the pose (position and/or rotation) of the user's eye and,based on this pose, shift a magnifier lens relative to the displayscreen of the near-eye display system so as to change the eye'sperception of the depth cues. As an example, conventional near-eyedisplay systems sometimes encounter a vergence-accommodation conflictdue to a mismatch between the vergence distance to focus on a virtualobject and the accommodation distance due to, for example, the near-eyeproximity of the display screen. The near-eye display system compensatesfor this vergence-accommodation conflict by shifting the magnifier lensrelative to the user's eye to cause a corresponding shift in theaccommodation plane in VR. By matching the shifted accommodation planeto the vergence plane, the near-eye display system is able to simulatereal-life viewing responses for both vergence and accommodation.

As another example, conventional near-eye display systems sometimesencounter gaze-dependent aberrations. The near-eye display system shiftsthe magnifier lens relative to the user's eye to correct for aberrationscaused by, for example, visual distortions due to gazing at differentportions of the display screen, irreproducibility in placing a VR/AR HMDat the same position on the user's face over repeated usages,differences in eye geometry (e.g., inter-pupillary distance) amongstdifferent users, etc. Thus, dynamically shifting the magnifier lens andrendering the image array responsive to shifts in the pose of the eye inthe user effectively compensates for verge-accommodation conflict andgaze-dependent aberration without sacrificing the focusing power of anyutilized optical components, which may otherwise lead to a correspondingreduction in the FOV and/or display frame-rate of the near-eye displaysystem.

FIG. 1 illustrates a near-eye display system 100 incorporating dynamicadjustments to compensate for vergence-accommodation disparities andgaze-dependent aberrations in accordance with at least one embodiment.In the depicted example, the near-eye display system 100 includes acomputational display sub-system 102, a rendering component 104, and oneor more eye tracking components, such as one or both of an eye trackingcomponent 106 for tracking a user's left eye and an eye trackingcomponent 108 for tracking the user's right eye. The computationaldisplay sub-system 102 includes a left-eye display 110 and a right-eyedisplay 112 mounted in an apparatus 114 (e.g., goggles, glasses, etc.)that places the displays 110, 112 in front of the left and right eyes,respectively, of the user.

As shown by view 116, each of the displays 110, 112 includes at leastone display panel 118 to display a sequence or succession of near-eyelightfield frames (hereinafter, “lightfield frame” for ease ofreference), each of which comprises an array 120 of elemental images122. For ease of reference, an array 120 of elemental images 122 mayalso be referred to herein as a lightfield frame 120. In someembodiments, a separate display panel 118 is implemented for each of thedisplays 110, 112, whereas in other embodiments the left-eye display 110and the right-eye display 112 share a single display panel 118, with theleft half of the display panel 118 used for the left-eye display 110 andthe right half of the display panel 118 used for the right-eye display112.

Each of the displays 110, 112 further includes one or more opticalelements 124, such as lenses, disposed to be overlaying the displaypanel 118. Cross-view 128 of FIG. 1 depicts a cross-section view alongline A-A of the optical elements 124 overlying the display panel 118such that the optical elements 124 overlies the display surface 130 ofthe display panel 118 so as to be disposed between the display surface130 and the corresponding eyes 132 of the user. In some embodiments,such as illustrated in FIG. 1, the optical elements 124 includemagnifying lenses disposed over each of the left-eye display 110 and theright-eye display 112. In this configuration, each optical element 124focuses a corresponding region of the display surface 130 onto the pupil134 of the eye.

Although depicted in this example as having a single magnifying lensdisposed over the displays 110, 112, the optical elements 124 can alsoinclude a plurality of lenses. For example, in some embodiments, theoptical elements 124 include (in addition to or in place of themagnifying lenses) an array of lenslets (not shown) overlying thedisplay surface 130 of the display panel 118 so as to be disposedbetween the display surface 130 and each corresponding eye 132 of theuser. In such configurations, each lenslet of the lenslet array focusesa corresponding region of the display surface 130 onto the pupil 134 ofthe eye, with each such region at least partially overlapping with oneor more adjacent regions. Thus, in such computational displayconfigurations, when an array 120 of elemental images 122 is displayedat the display surface 130 of the display panel 118 and then viewed bythe eye 132 through the lenslet array, the user perceives the array 120of elemental images 122 as a single image of a scene. Thus, when thisprocess is performed in parallel for both the left eye and right eye ofthe user with the proper parallax implemented therebetween, the resultis the presentation of autostereoscopic three-dimensional (3D) imageryto the user with the relatively wide FOV and shallow form factor oftenprovided by such computational displays.

Each of the optical elements 124 (e.g., magnifying lenses) is coupled toan actuator 126 configured to move the optical elements 124 along theX-axis, Y-axis, and Z-axis directions relative to the eyes 132 of theuser and the display surface 130 of the display panel 118. In variousembodiments, the actuator 126 is controlled by the rendering component104 to change the physical distance between the display panel 118 andthe eyes 132 of the user. For example, the actuator 126 may includeoptomechanical actuators such as piezo-electric, voice-coil, orelectro-active polymer actuators. Although described here in the contextof optomechanical actuators, those skilled in the art will recognizethat any mechanical actuator capable of physically moving the opticalelements 124 relative to the display panel 118 and the eyes 132 of theuser may be used without departing from the scope of this disclosure.

As also shown in FIG. 1, the rendering component 104 includes a set ofone or more processors, such as the illustrated central processing unit(CPU) 136 and graphics processing units (GPUs) 138, 140 and one or morestorage components, such as system memory 142, to store softwareprograms or other executable instructions that are accessed and executedby the processors 136, 138, 140 so as to manipulate the one or more ofthe processors 136, 138, 140 to perform various tasks as describedherein. Such software programs include, for example, rendering program144 comprising executable instructions for a lightfield frame renderingprocess, as described below, as well as an eye tracking program 146comprising executable instructions for an eye tracking process, as alsodescribed below.

In operation, the rendering component 104 receives rendering information148 from a local or remote content source 150, where the renderinginformation 148 represents graphics data, video data, or other datarepresentative of an object or scene that is the subject of imagery tobe rendered and displayed at the display sub-system 102. Executing therendering program 144, the CPU 136 uses the rendering information 148 tosend drawing instructions to the GPUs 138, 140, which in turn utilizethe drawing instructions to render, in parallel, a series of lightfieldframes 151 for display at the left-eye display 110 and a series oflightfield frames 153 for display at the right-eye display 112 using anyof a variety of well-known VR/AR computational/lightfield renderingprocesses. As part of this rendering process, the CPU 136 may receivepose information 150 from an inertial management unit (IMU) 154, wherebythe pose information 150 is representative of a pose of the displaysub-system 102 and control the rendering of one or more pairs oflightfield frames 151, 153 to reflect the viewpoint of the object orscene from the pose.

As described in detail below, the rendering component 104 further mayuse eye pose information from one or both of the eye tracking components106, 108 to shift the position of the magnifier lenses 124, and therebychanging the perception of the lightfield frame by the user's eyes 132.To this end, the eye tracking components 106, 108 each may include oneor more infrared (IR) light sources (referred to herein as “IRilluminators) to illuminate the corresponding eye with IR light, one ormore imaging cameras to capture the IR light reflected off of thecorresponding eye as a corresponding eye image (eye image information156), one or more mirrors, waveguides, beam splitters, and the like, todirect the reflected IR light to the imaging cameras, and one or moreprocessors to execute the eye tracking program 146 so as to determine acurrent position, current orientation, or both (singularly orcollectively referred to herein as “pose”) of the corresponding eye fromthe captured eye image. Any of a variety of well-known eye trackingapparatuses and techniques may be employed as the eye trackingcomponents 146, 148 to track one or both eyes of the user.

In at least one embodiment, the near-eye display system 100 maydetermine the eye pose as a past eye pose, a current eye pose, or apredicted (future) eye pose, or a combination thereof. In particular, aprediction of a future eye pose may provide improved performance orresponse time, and any of a variety of eye-movement predictionalgorithms may be implemented to predict a future eye pose. Moreover, insome instances, the eye-tracking components 106, 108 may utilize sceneinformation (e.g., location of faces within the imagery to be renderedor saliency heuristics) as input in prediction of a future gaze of theuser's eyes for eye pose calculation. As such, the term “eye pose”, asused herein, may refer to a previous, current, or predicted eye pose, orsome combination thereof.

In a conventional computational display-based near-eye system, avergence-accommodation conflict is sometimes encountered in which thereis a mismatch between the vergence distance to focus on a virtual objectand the accommodation distance due to, for example, the near-eyeproximity of the display panel. Additionally, the conventionalcomputational display-based near-eye system sometimes encountersgaze-dependent aberrations. As a result, the user's perception of thedisplayed imagery often is impacted, resulting in distorted views of thedisplayed imagery and discomfort/fatigue for the user.

As described herein, in at least one embodiment the near-eye displaysystem 100 mitigates the impact of vergence-accommodation conflicts byshifting the magnifier lenses 124 relative to the user's eye to cause acorresponding change in the accommodation distance (e.g., shift in theaccommodation plane in VR). This is accomplished by using the eyetracking components 106, 108 to track one or both eyes of the user so asto determine the pose of one or both of the eyes for a correspondinglightfield frame to be displayed. With the pose determined, theactuators 126 then shifts the positions of the magnifier lenses 124relative to the eyes 132. This shift in the position of the magnifierlenses 124 relative to the eyes 132 has the effect of shifting theaccommodation plane at which objects appear in focus. The renderingcomponent 104 then re-renders the elemental images 122 within thelightfield frame with a match between the shifted accommodation planeand the vergence plane containing the object focused on by the user'sgaze as determined by the pose of one or both of the eyes 132. Inaddition to shifting the accommodation plane, the rendering component104 may scale the dimensions of the displayed imagery on display panel118 so as to attempt to maintain a consistent size of the representedimagery in the virtual plane irrespective of Z-axis position of themagnifying lenses 124. In this manner, the position of the accommodationplane may be dynamically adjusted to accommodate the pose of the user'seye and better simulate real-life viewing responses for both vergenceand accommodation. Further, in at least one embodiment the near-eyedisplay system 100 mitigates the impact of gaze-dependent aberrations byshifting the magnifier lenses 124 relative to the user's eye 132. Thiscompensates for aberrations caused by, for example, visual distortionsdue to gazing at different portions of the display screen,irreproducibility in placing a VR/AR HIVID at the same position on theuser's face over repeated usages, differences in eye geometry (e.g.,inter-pupillary distance) amongst different users, and the like.

FIG. 2 illustrates a cross-section view of a computational display suchas the ones utilized in the near-eye display system 100 usingopto-mechanical actuators for dynamic correction ofvergence-accommodation conflicts in accordance with some embodiments. Asshown in the diagram 200 of the cross-section view, the eyes 132 of theuser are directed towards a point 202 in the stereoscopic object plane204 positioned at virtual depth d₁. As previously discussed relative toFIG. 1, each of the magnifier lenses 124 focuses a corresponding regionof the display surface 130 onto the pupil 134 of the eye 132. Further,the magnifier lenses 124 are each coupled to an actuator 126 configuredto move the magnifier lenses 124 along the X-axis, Y-axis, and Z-axisdirections relative to the eyes 132 of the user and the display panel118. The computational display also includes one or more eye trackingcomponents, such as one or both of an eye tracking component 106 fortracking the user's left eye and an eye tracking component 108 fortracking the user's right eye.

The virtual image is rendered for display such that an object at point202 in the virtual image (and/or other objects at the same virtual imageplane 208 as point 202) appears to be in focus when the eyes 132 of theuser accommodates and changes optical power to have a focal distanceapproximating that of the illustrated accommodation distance d₂. Pointscorresponding to that focal plane (i.e., virtual image plane 208) withinthe virtual image appear to be in-focus while points corresponding toother virtual image planes become blurred. Those skilled in the art willrecognize that the focal plane may also be referred to as the“accommodation plane,” as is used interchangeably herein.

In real-life viewing (as opposed to viewing a virtual scene), vergenceand accommodation are consistent with one another: looking at a nearerobject requires convergence and an increase in lens focal power, whilelooking at a farther object requires divergence and a decrease in focalpower. Because the distances to which the eyes converge and accommodateare generally the same, the two responses are coupled such that changesin vergence produce changes in accommodation, and vice versa.Additionally, retinal blur varies consistently with changes in scenedepth in real-life viewing, in which the perceived retinal image (atretinas of the eyes 132) is sharpest for objects at the distance towhich the eye is focused and blurred for nearer and farther objects. Asthe eyes 132 look around a real-life scene, neural commands are sent tothe lens muscles of the eyes 132 to change focal power and therebyminimize blur for the fixated part of the scene to provide depth cues.The correlation between blur and depth in real-life scenes aids depthperception.

With computational displays, as the eyes 132 look around a simulatedvirtual scene, the focal distance of the light originating from displaypanel 118 generally does not vary. Accordingly, focus cues (e.g.,accommodation and blur in the retinal image) specify the depth of thedisplay rather than the depths of objects in the simulated virtualscene. However, vergence stimulus varies depending on where the userlooks in the simulated virtual scene. Eye vergence varies to fixate ondifferent objects at different virtual depth in the simulated virtualscene. Accordingly, computational displays can encountervergence-accommodation conflicts when the vergence distance d₁ (e.g.,virtual depth of object from stereoscopic view, such as point 202)differs from the accommodation distance d₂.

For example, as illustrated in FIG. 2, a disparity d₃ exists between thevergence distance d₁ and the accommodation distance d₂ due to thevirtual depth of the point 202 being positioned further away than theaccommodation plane 208 at which the eyes 132 perceive objects as beingin focus. Similarly, a disparity can arise when the virtual depth ofanother point (not shown) is positioned closer than the accommodationplane 208 at which the eyes 132 perceive objects as being in focus. Adifference in those distances requires the user to uncouple vergence andaccommodation, in which the accommodation distance relative to variabledepths of virtual objects results in a binocular disparity. Theuncoupling of vergence and accommodation often reduces viewer ability tofuse binocular stimulus, resulting in discomfort and fatigue for theuser.

The disparity between the vergence plane and the accommodation planes ofthe eyes can be decreased using varifocal mechanisms to dynamicallycorrect for vergence-accommodation conflicts. As shown in the diagram206 of the cross-section view, the position of one or both of themagnifier lenses 124 may be adjusted by the actuators 126 to shift theaccommodation plane 208 closer to the stereoscopic object plane 204(i.e., vergergence plane) to decrease or eliminate the disparity d₃. Theeye tracking components 106, 108 track the position of the pupils 134 ofthe eye 132 so as to determine the pose of one or both of the eyes 132(e.g., the direction of gaze of the eye).

As described herein, the dynamic magnifier lens translation andaccommodation plane adjustment process utilizes an eye trackingcomponent (e.g., eye tracking components 106, 108) to determine the poseof a corresponding eye. This eye tracking component typically includesone or more IR illuminators to illuminate the eye, an imaging camera tocapture imagery of IR reflections from the eye, one or more lenses,waveguides, or other optical elements to guide the reflected IR lightfrom the eye to the imaging camera, and one or more processors executinga software program to analyze the captured imagery. Although the eyetracking components 106, 108 are illustrated as being overlaying thedisplay panel 118 such as to be disposed between the magnifier lensesand the eyes 132 of the user (and therefore provide a benefit ofproviding a less oblique or more direct viewpoint of the pupil andcornea of the eye, and thus facilitating improved eye trackingaccuracy), those skilled in the art will recognize that any positioningof the eye tracking components 106, 108 within the near-eye displaysystem 100 of FIG. 1 may be utilized without departing from the scope ofthis disclosure.

In the example of FIG. 2, the eye tracking components 106, 108 determinethat the eyes 132 of the user are directed towards a point 202 in thestereoscopic object plane 204 positioned at virtual depth d₁. To correctfor the disparity d₃ between the vergence plane 204 (i.e., stereoscopicobject plane) and the accommodation plane 208 (i.e., virtual imageplane), a magnifier lens shift may be computed (e.g., by renderingcomponent 104 of FIG. 1) to shift the position of at least one of themagnifier lenses 124 relative to the eyes 132 and the display panel 118.

As illustrated in the diagram 206 of the cross-section view, theactuators 126 have translated the magnifier lenses 124 along the Z-axisdirection to be positioned closer to the eyes 132 and further away fromthe display panel 118. This shifts the accommodation plane 208 furtheraway from the eyes 132 (i.e., increases the accommodation distance d₂).In contrast, actuating the actuators 126 to translate the magnifierlenses 124 along the Z-axis direction to be positioned further from theeyes 132 and closer to the display panel 118 shifts the accommodationplane 208 closer to the eyes 132 (i.e., decreases the accommodationdistance d₂). As illustrated in the diagram 206 of the cross-sectionview, shifting of the accommodation plane 208 further away from the eyes132 to match the distance of the vergence plane 204 allows the pupils134 of the eyes 132 to rotate away from each other (i.e., divergence)and fixate on the point 202 to simulate real-life viewing responses forboth vergence and accommodation.

Accordingly, the disparity between vergence distance d₁ and theaccommodation distance d₂ can be reduced by matching the vergence plane204 and accommodation plane 208. Although described here in the contextof a simultaneous shift of both magnifier lenses, those skilled in theart will recognize that each of the magnifier lenses 124 may beindependently actuated relative to each other and provide independentlight paths for the two eyes 132.

FIG. 3 is a method 300 of operation of the near-eye display system 100for rendering lightfield frames with adjusted magnifier lens positioningto provide dynamic vergence-accommodation disparity correction inaccordance with some embodiments. To facilitate understanding, method300 is described below with frequent reference to example scenariosillustrated by FIGS. 1-2. The method 300 illustrates one iteration ofthe process for rendering and displaying a lightfield frame for one ofthe left-eye display 110 or right-eye display 112, and thus theillustrated process is repeatedly performed in parallel for each of thedisplays 110, 112 to generate and display a different stream or sequenceof lightfield frames for each eye at different points in time, and thusprovide a 3D, autostereoscopic VR or AR experience to the user.

For a lightfield frame to be generated and displayed, method 300 startsat block 302, whereby the rendering component 104 identifies the imagecontent to be displayed to the corresponding eye of the user as alightfield frame. In at least one embodiment, the rendering component104 receives the IMU information 152 representing data from variouspose-related sensors, such as a gyroscope, accelerometer, magnetometer,Global Positioning System (GPS) sensor, and the like, and from the IMUinformation 150 determines a pose of the apparatus 114 (e.g., HMD) usedto mount the displays 110, 112 near the user's eyes. From this pose, theCPU 136, executing the rendering program 144, can determine acorresponding current viewpoint of the subject scene or object, and fromthis viewpoint and graphical and spatial descriptions of the scene orobject provided as rendering information 148, determine the imagery tobe rendered for the pose.

At block 304, the CPU 136, executing eye tracking program 146,determines the pose of the corresponding eye of the user. As explainedherein, the pose of an eye may be determined using any of a variety ofeye tracking techniques. Generally, such techniques include the captureof one or more images of IR light reflected from the pupil and cornea ofthe eye. The eye tracking program 146 then may manipulate the CPU 136 orthe GPUs 138, 140 to analyze the images to determine the pose of the eyebased on the corresponding position of one or both of the pupilreflection or corneal reflection. For example, in some embodiments,monocular eye tracking is performed to obtain region of interestinformation and calculate where an eye of the user is attempting toaccommodate in a rendered scene (e.g., which object(s) in a scene is thegaze of the eye directed towards). By performing monocular eye trackingfor each eye, a relative angular displacement between the two eyes ismeasured to determine vergence. Accordingly, accommodation is calculatedbased on the determined vergence (e.g., differential eye tracking). Inother embodiments, binocular eye tracking is performed to determineaccommodation independent of rendered scene content and/or orientationof the pupil relative to the cornea in turn may be used to determine theorientation of the eye (that is, the direction of gaze of the eye). Itshould be noted that although block 304 is illustrated in FIG. 3 asbeing subsequent to block 302, the process of block 304 may be performedbefore, during, or after the process of block 302.

With the pose of the user's eye determined, at block 306 the renderingprogram 144 manipulates the CPU 136 to determine and actuate a magnifierlens shift based on the pose of the user's eye. In some embodiments,determining the magnifier lens shift includes determining a shift vectorrepresenting desired translation of the magnifier lens inthree-dimensional (3D) space. As explained above, the magnifier lensshift represents a shift in position to be applied to the magnifierlenses so as to change the accommodation plane distance to the eye. Inparticular, the magnifier lens shift is intended to match the vergenceplane and accommodation plane for the eye based on the pose of theuser's eye. That is, the magnifier lens shift is to serve to dynamicallyreduce disparities between vergence distance and accommodation distancefor the eye.

In at least one embodiment, the determination of the magnifier lensshift is based on the calculation of a shift vector to be applied forchanging the distance between the magnifier lens relative to the user'seye and the display panel. To illustrate, referring to an examplescenario illustrated by cross-section view of FIG. 2, a pose for theuser's eye 132 relative to the display panel 118 is determined toidentify a point 202 in the virtual image at which the user's gaze isdirected. With this example, a disparity d₃ between the vergencedistance d₁ and the accommodation distance d₂ is identified.Accordingly, a shift vector is calculated and communicated to theactuator 126 to translate the magnifier lens 124 along the Z-axisdirection to match distances from the eye 132 to the vergence plane 204and accommodation plane 208 within the virtual image. Actuation of themagnifier lenses 124 to be positioned further from the eyes 132 (e.g.,closer to the display panel 118) shifts the accommodation plane 208closer to the eyes 132, and vice versa.

Referring back to FIG. 3, with the magnifier lens shift determined, andscaling of the elemental images determined, as appropriate, at block 308the rendering program 144 manipulates the CPU 136 to instruct thecorresponding one of the GPUs 138, 140 to render a lightfield frame witharray 120 using the image content identified at block 302, whereby thelightfield frame includes an array of elemental images. In someembodiments, as part of this process, the CPU 136 calculates a relativemagnification change based on the determined magnifier lens shift ofblock 306. In particular, there is a magnification dependence oneye-accommodation state and changing the eye accommodation distance mayrequire a scaling of the displayed image to maintain the same view ofthe image content of block 302. For example, in the context of FIG. 2,the increased accommodation distance of the diagram 206 requires amagnification of an image rendered on the display panel 118 to maintainthe same view of the virtual image as experienced by the eyes in thediagram 200 of FIG. 2. Accordingly, the CPU 136 provides the magnifierlens shift information and indication of any scaling to be applied tothe dimensions of the elemental images to the GPU and instructs the GPUto render the lightfield frame such that the elemental images arerendered with matching vergence and accommodation planes, and scaled inaccordance with the supplied scaling information, if any. The GPUsubsequently renders the lightfield frame at block 310 and provides thelightfield frame to the corresponding one of the computational displays110, 112 for display to the eye 132 of the user with the adjustment tothe accommodation plane and/or relative magnification change of blocks306 and 308.

The display of AR/VR imagery is sometimes associated with gaze-dependentaberrations, such as pupil swim when the eye's perception of displayedimagery distorts as the eye moves. Typically, distortion (as motion,etc.) is experienced as the eyes rotate off its center axis, andespecially towards the edge of the lens in VR, as the distortion fieldat the edge is different than the distortion field at the center. FIG. 4illustrates a cross-section view of a computational display such as theones utilized in the near-eye display system 100 using opto-mechanicalactuators for dynamic correction of gaze-dependent aberrations inaccordance with some embodiments. Similar to the example embodimentsdiscussed above relative to FIGS. 2-3, the eyes 132 of the user aredirected towards a point 402 in the stereoscopic object plane 404positioned at virtual depth d₁. Each of the magnifier lenses 124 focusesa corresponding region of the display surface 130 onto the pupil 134 ofthe eye 132. Further, the magnifier lenses 124 are each coupled to anactuator 126 configured to move the magnifier lenses 124 along theX-axis, Y-axis, and Z-axis directions relative to the eyes 132 of theuser and the display panel 118. The computational display also includesone or more eye tracking components, such as one or both of an eyetracking component 106 for tracking the user's left eye and an eyetracking component 108 for tracking the user's right eye.

The virtual image is rendered for display such that an object at point402 in the virtual image (and/or other objects at the same virtual imageplane 408 as point 402) appears to be in focus when the eyes 132 of theuser accommodates and changes optical power to have a focal distanceapproximating that of the illustrated accommodation distance d₂. Pointscorresponding to that focal plane (i.e., virtual image plane 408) withinthe virtual image appear to be in focus while points corresponding toother virtual image planes become blurred. Those skilled in the art willrecognize that the focal plane may also be referred to as the“accommodation plane,” as is used interchangeably herein.

As illustrated in FIG. 4, a disparity exists between the vergencedistance and the accommodation distance due to the virtual depth of thepoint 402 being positioned further away than the accommodation plane 408at which the eyes 132 perceive objects as being in focus. Similarly, adisparity can arise when the virtual depth of another point (not shown)is positioned closer than the accommodation plane 408 at which the eyes132 perceive objects as being in focus. The disparity between thevergence plane and the accommodation planes of the eyes can be decreasedusing varifocal mechanisms to dynamically correct forvergence-accommodation conflicts, such as described in more detailabove. As shown in the diagram 406 of the cross-section view, theposition of one or both of the magnifier lenses 124 may be adjusted bythe actuators 126 to shift the accommodation plane 408 closer to thestereoscopic object plane 404 (i.e., vergence plane) based on the poseof one or both of the eyes 132 (e.g., the direction of gaze of the eye)using the eye tracking components 106, 108 track the position of thepupils 134 of the eye 132.

As described herein, the dynamic magnifier lens translation,accommodation plane adjustment, and gaze-dependent aberration correctionprocesses utilizes an eye tracking component (e.g., eye trackingcomponents 106, 108) to determine the pose of a corresponding eye. Thiseye tracking component typically includes one or more IR illuminators toilluminate the eye, an imaging camera to capture imagery of IRreflections from the eye, one or more lenses, waveguides, or otheroptical elements to guide the reflected IR light from the eye to theimaging camera, and one or more processors executing a software programto analyze the captured imagery. Although the eye tracking components106, 108 are illustrated as being overlaying the display panel 118 suchas to be disposed between the magnifier lenses and the eyes 132 of theuser, those skilled in the art will recognize that any positioning ofthe eye tracking components 106, 108 within the near-eye display system100 of FIG. 1 may be utilized without departing from the scope of thisdisclosure.

The computational display of FIG. 4 shifts the magnifier lenses 124 toaccount for gaze-dependent aberrations, such as pupil swim, orvariations in eye geometry amongst different users. As illustrated inthe diagram 406 of the cross-section view, the actuators 126 havetranslated the magnifier lenses 124 along the Z-axis direction to bepositioned closer to the eyes 132 and further away from the displaypanel 118. This shifts the accommodation plane 408 further away from theeyes 132 to match the distance of the vergence plane 404, thussimulating real-life viewing responses for both vergence andaccommodation. Further, the actuators 126 have translated the magnifierlenses 124 along at least one of the X- and Y-axis directions to correctfor gaze-dependent aberrations. For example, as illustrated in thediagram 406 of the cross-section view, the actuators 126 have furthertranslated the magnifier lenses 124 (in addition to the Z-axistranslation) along X-axis (and/or Y-axis) direction to correct forgaze-dependent aberrations, such as changes in the distortion field dueto the off-center axis position of point 402 (relative to the center ofdisplay eye focus on point 202 of FIG. 2). Further, the actuators 126may translate the magnifier lenses 124 to account for differential eyegeometry amongst various users (e.g., inter-pupillary distance, etc.).Further, in some embodiments, the actuators 126 may translate themagnifier lenses 124 to account for differential geometry between thepositioning of the near-eye display system 100 on the user's headrelative to the eyes 132 across different viewing sessions and/orpositioning of the near-eye display system 100 on the heads of varioususers. Accordingly, gaze-dependent aberrations may be dynamicallycorrected for based on the pose of a user's eye(s) relative to thedisplay screen. Although described here in the context of a simultaneousshift of both magnifier lenses, those skilled in the art will recognizethat each of the magnifier lenses 124 may be independently actuatedrelative to each other and provide independent light paths for the twoeyes 132. Further, although described here in the context of a shift ofthe magnifier lenses to correct for both vergence-accommodationdisparities and gaze-dependent aberrations, those skilled in the artwill recognize that the gaze-dependent aberration correction may beapplied (e.g., X- and/or Y-axis shifts as described herein)independently of the vergence-accommodation conflict correction (e.g.,Z-axis shift as described herein).

FIG. 5 is a method 500 of operation of the near-eye display system 100for rendering lightfield frames with adjusted magnifier lens positioningto provide dynamic gaze-dependent aberration correction in accordancewith some embodiments. To facilitate understanding, method 500 isdescribed below with frequent reference to example scenarios illustratedby FIGS. 1-4. The method 500 illustrates one iteration of the processfor rendering and displaying a lightfield frame for one of the left-eyedisplay 110 or right-eye display 112, and thus the illustrated processis repeatedly performed in parallel for each of the displays 110, 112 togenerate and display a different stream or sequence of lightfield framesfor each eye at different points in time, and thus provide a 3D,autostereoscopic VR or AR experience to the user.

For a lightfield frame to be generated and displayed, method 500 startsat block 502, whereby the rendering component 104 identifies the imagecontent to be displayed to the corresponding eye of the user as alightfield frame. In at least one embodiment, the rendering component104 receives the IMU information 152 representing data from variouspose-related sensors, such as a gyroscope, accelerometer, magnetometer,Global Positioning System (GPS) sensor, and the like, and from the IMUinformation 150 determines a pose of the apparatus 114 (e.g., HMD) usedto mount the displays 110, 112 near the user's eyes. From this pose, theCPU 136, executing the rendering program 144, can determine acorresponding current viewpoint of the subject scene or object, and fromthis viewpoint and graphical and spatial descriptions of the scene orobject provided as rendering information 148, determine the imagery tobe rendered for the pose.

At block 504, the CPU 136, executing eye tracking program 146,determines the pose of the corresponding eye of the user. As explainedherein, the pose of an eye may be determined using any of a variety ofeye tracking techniques. Generally, such techniques include the captureof one or more images of IR light reflected from the pupil and cornea ofthe eye. The eye tracking program 146 then may manipulate the CPU 136 orthe GPUs 138, 140 to analyze the images to determine the pose of the eyebased on the corresponding position of one or both of the pupilreflection or corneal reflection. For example, in some embodiments,monocular eye tracking is performed to obtain region of interestinformation and calculate where an eye of the user is attempting toaccommodate in a rendered scene (e.g., which object(s) in a scene is thegaze of the eye directed towards). By performing monocular eye trackingfor each eye, a relative angular displacement between the two eyes ismeasured to determine vergence. Accordingly, accommodation is calculatedbased on the determined vergence (e.g., differential eye tracking). Inother embodiments, binocular eye tracking is performed to determineaccommodation independent of rendered scene content and/or orientationof the pupil relative to the cornea in turn may be used to determine theorientation of the eye (that is, the direction of gaze of the eye). Itshould be noted that although block 504 is illustrated in FIG. 5 asbeing subsequent to block 502, the process of block 504 may be performedbefore, during, or after the process of block 502.

With the pose of the user's eye determined, at block 506 the renderingprogram 144 manipulates the CPU 136 to determine and actuate a magnifierlens shift based on the pose of the user's eye. In some embodiments,determining the magnifier lens shift includes determining a shift vectorrepresenting desired translation of the magnifier lens inthree-dimensional (3D) space. As explained above, the magnifier lensshift represents a shift in position to be applied to the magnifierlenses so as to dynamically correct for gaze-dependent aberrationsand/or correct for differential positioning of a user's eye(s) relativeto the display screen.

In at least one embodiment, the determination of the magnifier lensshift is based on the calculation of a shift vector to be applied forshifting the magnifier lenses as the user's eye moves around to look atdifferent portions of a virtual image. To illustrate, referring to theexample scenario illustrated by cross-section view of FIG. 4, a pose forthe user's eye 132 relative to the display panel 118 is determined toidentify a point 402 in the virtual image at which the user's gaze isdirected. The actuators 126 have translated the magnifier lenses 124along at least one of the X- and Y-axis directions to correct forgaze-dependent aberrations, such as changes in the distortion field dueto the off-center axis position of point 402 (relative to the center ofdisplay eye focus on point 202 of FIG. 2).

In other embodiments, the determination of the magnifier lens shift isbased on the calculation of a shift vector to be applied for shiftingthe magnifier lenses to account for differential eye geometry amongstvarious users (e.g., inter-pupillary distance, etc.). For example, theactuators 126 may translate the magnifier lenses 124 to be positionedcloser (or further) away from each other to account for one user havinga narrower (or wider) IPD from another user. Similarly, in otherembodiments, the actuators 126 may translate the magnifier lenses 124 toaccount for differential geometry between the positioning of thenear-eye display system 100 on the user's head relative to the eyes 132across different viewing sessions and/or positioning of the near-eyedisplay system 100 on the heads of various users. For example, it isoften difficult for each individual user to reproducibly position thenear-eye display system 100 at the same position on his/her face acrossmultiple viewing sessions (e.g., taking HMD on and off).

In other embodiments, the determination of the magnifier lens shift mayfurther include calculating the shift vector to change the distancebetween the magnifier lens relative to the user's eye and the displaypanel. For example, as illustrated in the diagram 406 of thecross-section view, the actuators 126 have also translated the magnifierlenses 124 along the Z-axis direction for vergence-accommodationconflict correction in addition to the X- and/or Y-axis translation forgaze-dependent aberration correction.

Accordingly, at block 506, a shift vector is calculated and communicatedto the actuator 126 to translate the magnifier lens 124 along at leastone of the X-, Y-, and Z-axis directions to perform gaze-dependentaberration correction. With the magnifier lens shift determined, andscaling of the elemental images determined, as appropriate, at block 508the rendering program 144 manipulates the CPU 136 to instruct thecorresponding one of the GPUs 138, 140 to render a lightfield frame witharray 120 using the image content identified at block 502, whereby thelightfield frame includes an array of elemental images.

In some embodiments, as part of this process, the CPU 136 calculates arelative magnification change based on the determined magnifier lensshift of block 506. In particular, there is a magnification dependenceon eye-accommodation state and changing the eye accommodation distancemay require a scaling of the displayed image to maintain the same viewof the image content of block 502 if the accommodation plane has beenchanged. For example, in the context of FIG. 2, the increasedaccommodation distance of the diagram 206 requires a magnification of animage rendered on the display panel 118 to maintain the same view of thevirtual image as experienced by the eyes in the diagram 200 of FIG. 2.

The CPU 136 provides the magnifier lens shift information and indicationof any scaling to be applied to the dimensions of the elemental imagesto the GPU and instructs the GPU to render the lightfield frame suchthat the elemental images are rendered to account for gaze-dependentaberrations. Further, in some embodiments, the GPU is instructed torender the lightfield frame such that the elemental images are renderedto for matching vergence and accommodation planes, and scaled inaccordance with the supplied scaling information, if any. The GPUsubsequently renders the lightfield frame at block 510 and provides thelightfield frame to the corresponding one of the computational displays110, 112 for display to the eye 132 of the user.

The eye tracking and optical component configurations illustrated byFIGS. 1-5 have the benefit of providing dynamic correction forvergence-accommodation disparities and gaze-dependent aberrations whileretaining a large field-of-view, high spatial-resolution, and a highframe-rate. That is, the embodiments described herein allow forvergence-accommodation disparity and/or gaze-dependent aberrationcorrection without sacrificing the focusing power of the utilizedoptical components, which may otherwise lead to a correspondingreduction in the FOV and/or display frame-rate of the near-eye displaysystem. For example, the embodiments described herein do not require anytemporal multiplexing of signals generated by the near-eye displaysystem, reduce the native display frame-rate of display panel, or placeany intervening optics that reduce light throughput from the displaypanel to the user's eyes.

In some embodiments, certain aspects of the techniques described abovemay implemented by one or more processors of a processing systemexecuting software. The software comprises one or more sets ofexecutable instructions stored or otherwise tangibly embodied on anon-transitory computer readable storage medium. The software caninclude the instructions and certain data that, when executed by the oneor more processors, manipulate the one or more processors to perform oneor more aspects of the techniques described above. The non-transitorycomputer readable storage medium can include, for example, a magnetic oroptical disk storage device, solid state storage devices such as Flashmemory, a cache, random access memory (RAM) or other non-volatile memorydevice or devices, and the like. The executable instructions stored onthe non-transitory computer readable storage medium may be in sourcecode, assembly language code, object code, or other instruction formatthat is interpreted or otherwise executable by one or more processors.

A computer readable storage medium may include any storage medium, orcombination of storage media, accessible by a computer system during useto provide instructions and/or data to the computer system. Such storagemedia can include, but is not limited to, optical media (e.g., compactdisc (CD), digital versatile disc (DVD), Blu-Ray disc), magnetic media(e.g., floppy disc, magnetic tape, or magnetic hard drive), volatilememory (e.g., random access memory (RAM) or cache), non-volatile memory(e.g., read-only memory (ROM) or Flash memory), ormicroelectromechanical systems (MEMS)-based storage media. The computerreadable storage medium may be embedded in the computing system (e.g.,system RAM or ROM), fixedly attached to the computing system (e.g., amagnetic hard drive), removably attached to the computing system (e.g.,an optical disc or Universal Serial Bus (USB)-based Flash memory), orcoupled to the computer system via a wired or wireless network (e.g.,network accessible storage (NAS)).

Note that not all of the activities or elements described above in thegeneral description are required, that a portion of a specific activityor device may not be required, and that one or more further activitiesmay be performed, or elements included, in addition to those described.Still further, the order in which activities are listed are notnecessarily the order in which they are performed. Also, the conceptshave been described with reference to specific embodiments. However, oneof ordinary skill in the art appreciates that various modifications andchanges can be made without departing from the scope of the presentdisclosure as set forth in the claims below. Accordingly, thespecification and figures are to be regarded in an illustrative ratherthan a restrictive sense, and all such modifications are intended to beincluded within the scope of the present disclosure.

Benefits, other advantages, and solutions to problems have beendescribed above with regard to specific embodiments. However, thebenefits, advantages, solutions to problems, and any feature(s) that maycause any benefit, advantage, or solution to occur or become morepronounced are not to be construed as a critical, required, or essentialfeature of any or all the claims. Moreover, the particular embodimentsdisclosed above are illustrative only, as the disclosed subject mattermay be modified and practiced in different but equivalent mannersapparent to those skilled in the art having the benefit of the teachingsherein. No limitations are intended to the details of construction ordesign herein shown, other than as described in the claims below. It istherefore evident that the particular embodiments disclosed above may bealtered or modified and all such variations are considered within thescope of the disclosed subject matter. Accordingly, the protectionsought herein is as set forth in the claims below.

What is claimed is:
 1. In a near-eye display system, a methodcomprising: determining, using an eye tracking component of the near-eyedisplay system, a pose of a user's eye; determining a shift vector for amagnifier lens of the near-eye display system based on the pose of theuser's eye; communicating the shift vector to an actuator of thenear-eye display system to instruct translation of the magnifier lensrelative to the user's eye; rendering an array of elemental images at aposition within a near-eye lightfield frame; and communicating thenear-eye lightfield frame for display at a display panel of the near-eyedisplay system.
 2. The method of claim 1, wherein determining the shiftvector comprises: determining a vergence distance and an accommodationdistance to a target object focused on by the pose of the user's eye;and determining the shift vector to represent a shift in position of themagnifier lens in three-dimensional (3D) space to reduce a disparitybetween the vergence distance and the accommodation distance.
 3. Themethod of claim 2, wherein determining the shift vector furthercomprises: determining the shift vector as representing the shift inposition of the magnifier lens in 3D space to eliminate the disparitybetween the vergence distance and the accommodation distance.
 4. Themethod of claim 1, wherein determining the shift vector comprises:determining a vergence plane and an accommodation plane within thenear-eye lightfield frame based on the pose of the user's eye; anddetermining the shift vector as representing a shift in position of themagnifier lens in 3D space to converge the accommodation plane towardsthe vergence plane within the near-eye lightfield frame.
 5. The methodof claim 1, wherein determining the pose of the user's eye comprises:capturing imagery of the user's eye using an imaging camera disposedbetween the display panel and the user's eye.
 6. The method of claim 1,wherein determining the shift vector comprises: determining a viewingdistortion to be perceived based on the pose of the user's eye; anddetermining the shift vector as representing a shift in position of themagnifier lens in three-dimensional (3D) space to correct the viewingdistortion.
 7. The method of claim 6, wherein determining the shiftvector further comprises: identifying a gaze-dependent aberrationresulting from changes in a distortion field when the pose of the user'seye is redirected relative to the display panel.
 8. In a near-eyedisplay system, a method comprising: determining a pose of a first eyerelative to a display panel of the near-eye display system using an eyetracking component of the near-eye display system; rendering a firstnear-eye lightfield frame based on the pose of the first eye; detectinga change in position of the first eye along an axis perpendicular to thedisplay panel; determining a viewing distortion to be perceived based onthe change in position of the first eye; determining a shift vector fora magnifier lens of the near-eye display system based on the pose of thefirst eye, the shift vector representing a shift in position of themagnifier lens in three-dimensional space (3D) to correct the viewingdistortion; actuating the magnifier lens based on the shift vector; andrendering and displaying a second near-eye lightfield frame based on theshift vector.
 9. The method of claim 8, wherein determining the shiftvector comprises: determining a vergence plane and an accommodationplane within the first near-eye lightfield frame based on the pose ofthe first eye; and determining the shift vector as representing a shiftin position of the magnifier lens in 3D space to converge theaccommodation plane towards the vergence plane within the secondnear-eye lightfield frame.
 10. The method of claim 9, furthercomprising: determining a scaling factor based on the shift vector; andscaling a display size of the second near-eye lightfield frame on thedisplay panel based on the scaling factor to maintain a same perceivedview of the second near-eye lightfield frame relative to the firstnear-eye lightfield frame.
 11. The method of claim 8, whereindetermining the shift vector further comprises: identifying agaze-dependent aberration resulting from changes in a distortion fielddue to the change in position of the first user's eye.
 12. The method ofclaim 8, further comprising: determining a pose of a second eye relativeto the display panel; determining a second shift vector for themagnifier lens based on the pose of the second eye; actuating themagnifier lens based on the second shift vector; and rendering anddisplaying a third near-eye lightfield frame on the display panel basedon the second shift vector to maintain a same perceived view of thethird near-eye lightfield frame by the second eye relative to the secondnear-eye lightfield frame by the first eye.
 13. A near-eye displaysystem comprising: a display panel to display a near-eye lightfieldframe comprising an array of elemental images; an eye tracking componentto track a pose of a user's eye; an actuator to shift a position of amagnifier lens based on the pose of the user's eye; and a renderingcomponent to position the array of elemental images within the near-eyelightfield frame based on the pose of the user's eye and the shiftedposition of the magnifier lens, wherein the rendering component is toposition the array of elemental images by determining a viewingdistortion to be perceived based on the pose of the user's eye andrendering the near-eye lightfield frame based on the shifted position ofthe magnifier lens in 3D space to correct the viewing distortion. 14.The near-eye display system of claim 13, wherein the rendering componentis to position the array of elemental images within the near-eyelightfield frame by: determining a vergence plane and an accommodationplane within the near-eye lightfield frame based on the pose of theuser's eye; and rendering the near-eye lightfield frame based on theshifted position of the magnifier lens in 3D space to converge theaccommodation plane towards the vergence plane within the near-eyelightfield frame.
 15. The method of claim 13, wherein determining therendering component is further to position the array of elemental imageswithin the near-eye lightfield frame by: identifying a gaze-dependentaberration resulting from changes in a distortion field when a gaze ofthe user's eye is redirected on the display panel.
 16. A renderingsystem comprising: at least one processor; an input to receive data froman eye tracking component indicating a pose of a user's eye relative toa near-eye display panel, wherein the input is further to receive dataindicating a shifted position of a magnifier lens; and a storagecomponent to store a set of executable instructions, the set ofexecutable instructions configured to manipulate the at least oneprocessor to determine a vergence plane and an accommodation planewithin the near-eye lightfield frame based on the pose of the user'seye, the set of executable instructions further configured to manipulatethe at least one processor to render a near-eye lightfield framecomprising an array of elemental images having a position within thenear-eye lightfield frame based on the pose of the user's eye and theshifted position of the magnifier lens to converge the accommodationplane towards the vergence plane within the near-eye lightfield frame.17. The rendering system of claim 16, wherein the set of executableinstructions are configured to manipulate the at least one processor torender the near-eye lightfield frame by: determining a viewingdistortion to be perceived based on the pose of the user's eye; andrendering the near-eye lightfield frame based on the shifted position ofthe magnifier lens to correct the viewing distortion.
 18. In a near-eyedisplay system, a method comprising: determining, using an eye trackingcomponent of the near-eye display system, a pose of a user's eye;determining a vergence plane and an accommodation plane within anear-eye lightfield frame based on the pose of the user's eye;determining a shift vector for a magnifier lens of the near-eye displaysystem based on the pose of the user's eye, the shift vectorrepresenting a shift in position of the magnifier lens inthree-dimensional (3D) space to converge the accommodation plane towardsthe vergence plane within the near-eye lightfield frame; communicatingthe shift vector to an actuator of the near-eye display system toinstruct translation of the magnifier lens relative to the user's eye;rendering an array of elemental images at a position within the near-eyelightfield frame; and communicating the near-eye lightfield frame fordisplay at a display panel of the near-eye display system.